Skip to content

feat: returns optimization — cash sweep, exit guards, resume lane, feedback repair (forensics-driven) - #101

Merged
yebof merged 8 commits into
mainfrom
feat/returns-optimization
Jul 16, 2026
Merged

feat: returns optimization — cash sweep, exit guards, resume lane, feedback repair (forensics-driven)#101
yebof merged 8 commits into
mainfrom
feat/returns-optimization

Conversation

@yebof

@yebof yebof commented Jul 16, 2026

Copy link
Copy Markdown
Owner

Returns optimization: forensics-driven fixes for the July underperformance

Why (the forensics, 2026-07-16)

A 7-agent diagnostic workflow root-caused the gap (NAV −0.5% vs SPY +2.5% over the 6/5–7/14 validation window; −3.5% vs SPY since 5/1; 84% idle cash, no BUY since 6/26):

# Root cause $ impact (est.)
RC1 Exit engine whipsaws winners: trail-stops ratcheted into daily noise (LLY stopped twice identically, 5 trail fills missed avg +30.7% after exit), advisory to_stop treated as a trigger, same-day full exits on single news stories (AAPL +18% sold on one component-cost story) $8.5k–16k
RC2 Morning session mechanically KILLED at the PM→RM boundary by the wrapper timeout during the relay outage — 61/61 BUY-proposal days destroyed 6/30–7/15, zero RM vetoes, retry ticks re-burned full research to die identically $2–3k + unbounded
RC3 Chronic under-deployment: macro demanded 72–75% invested for 3 months, book averaged 39% and declined monotonically; nothing reconciled the compounded size-shaving against the target ~$8.9k cash drag
RC4 Self-grading loop is self-exculpatory: sell grades scored at t+1..t+3 (32/33 "correct" while the tape had 53% of exits ≥5% higher within 20d); value_entry_missed code-filtered out of PM's view (SNDK flagged 16×, PM never saw it); misses grouped by never-repeating free-text themes reinforces RC1/RC3
RC5 Observability blind: SIGKILL skips python's finally-block notifier (13 days of kills → zero pushes); the dead-man check accepted any run- row as "morning ran" turned 1 day into 13

A separate 25-agent optimization hunt proposed 20 improvements; 19 survived adversarial judging. The top-scored, invariant-safe subset is implemented here; the rest are documented below as backlog.

What's in (8 commits)

  1. feat(cash) SGOV idle-cash sweep — deterministic, zero-LLM. Excess cash above a 1% reserve parks into SGOV at every session bookend; released automatically before BUYs. Strict cash-equivalence contract: hidden from every LLM view, counted as cash by the risk filter, excluded from exposure math, stopless by design, first to liquidate in force_delever, SWEEP_* action names invisible to all grading/calibration consumers. Open-BUY-order holds subtracted before parking (unknowable → park nothing). ~$300/mo risk-free carry at current cash levels.
  2. feat(exits) deterministic exit-quality guards — TRAIL_STOP noise-band clamp (reject stops inside 1.25×ATR14 of price), ratchet cooldown (one accepted tighten per ~2 trading days), live broker stop in position facts (the stale-wide BUY-row stop kept feeding the ratchet), entry stop floor at 1×ATR with automatic size compensation. Hard-trigger citations bypass the clamps.
  3. feat(feedback) grading-loop repairvalue_entry_missed now a real miss; misses grouped by symbol AND theme (dual-key); a deterministic post-exit reality block (trades × live prices, no LLM) rides into the reviewer prompt with an escalation line when >50% of recent exits kept running; reviewer prompt: to_stop is advisory, single-source news on a >10% winner caps first-day action at REDUCE≤50%, think in ATR units.
  4. feat(pm) deployment-gap convergence — PMFacts renders a ⚠️ DEPLOYMENT GAP section (>15pp under macro target) that the PM prompt requires be answered in cash_target: close it or name a checkable blocker. The macro_exposure_deviation advisory is direction-aware (no more "consider scale_all_buys" on an UNDER-deployed book).
  5. feat(pipeline) decision checkpoint + resume lane — the PM's plan is persisted the moment it exists; a killed morning resumes next tick at the RiskStage (full preamble re-runs, RM always re-runs, consumed-before-execution gives at-most-once). Converts the RC2 death mode into delayed-by-one-tick execution and stops retry ticks from re-burning research.
  6. feat(obs) violent-death reporting — wrapper pushes Telegram on kill statuses (124/137/143) where python's notifier cannot run; optional HEALTHCHECKS_URL external dead-man ping; the evening dead-man check now detects partial mornings (research-only rows, unconsumed checkpoint).
  7. fix(cash) — self-review fixes: reviewer view truly excludes the vehicle; midday/close park bookend.
  8. fix(review) — 22 confirmed findings from the adversarial review applied: RM sweep view (the veto layer saw SGOV as an 84% position while PM saw cash), emergency×park oscillation, ex-div false trail cooldown, resume-lane bar rehydration, fail-closed checkpoint consumption, dead-man probe accuracy + notifier banner matching, sweep hygiene in evening/learning views, post-widening R/R recheck.

Hard invariants — untouched

cash_only / force_delever semantics, SELL protection discipline (SWEEP_SELL uses _submit_protected_sell + finalize like every other path), RM veto power (resume re-runs RM, rejected plans consumed), 3% breaker, caps, ET discipline, allocation_pct semantics. The reviewer/PM prompt edits are operator edits (the auto-evolution protection is unaffected).

Test plan

  • Full suite: 1310 passed (was 1251 before this branch; +59 new tests across 5 new test files)
  • Adversarial review workflow (5 lenses × verify): 34 agents, 29 raw findings → 22 confirmed (all fixed in fix(review) commit), 7 refuted
  • Watch first live sessions: morning (research→PM→RM→exec + sweep bookend), midday (reviewer without SGOV + park), and the first SWEEP_BUY fill on Alpaca paper
  • Optional: set HEALTHCHECKS_URL in .env to activate the external dead-man switch

Deferred backlog (judged KEEP, not implemented here)

Outcome-aware replay scoring (score replayed decisions against realized forward returns); SPY-trend + breadth deterministic regime block; conviction-tiered risk budget; re-entry radar for stopped-out names + deterministic re-entry tickets; memory-layer ablation in the replay harness; deployment-rate calibration persistence; sector relative-strength table; cluster-capacity pre-clamp; broker-side stop-out fills into evening grading; house-money pyramid adds; pre-market gap scan.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PQkESoSTYx2bCy7WYnTPXR

yebof and others added 8 commits July 16, 2026 15:29
…mand

2026-07-16 forensics: the account sat at ~84% idle cash for weeks (~$8.9k
cash drag over the quarter, ~$300/mo of forgone risk-free carry). This adds
a deterministic, zero-LLM sweep with a strict cash-equivalence contract:

- CashSweeper (src/execution/cash_sweep.py): park_excess() buys the vehicle
  with cash above a reserve, minus open-BUY-order holds (Alpaca's cash field
  doesn't subtract them — sweeping that cash would starve pending fills;
  unknowable holds → park nothing). fund_buys() releases exactly enough
  parked cash before the BUY phase, via the same _submit_protected_sell +
  _finalize_pending_protections discipline as every other SELL path.
- The vehicle is hidden from every LLM view (PM decision stage, position
  reviewer, evening builders) and its market value is credited as CASH in
  _filter_hard_risk_decisions — parked cash can never block a real BUY via
  the net-exposure/cash rules.
- force_delever sells it FIRST (tier -1); _reconcile_stop_coverage exempts
  it (deliberately stopless); SWEEP_BUY/SWEEP_SELL action names keep it out
  of every grading/calibration consumer.
- config: cash_sweep section (enabled/symbol/reserve_pct/min_order_usd),
  default-disabled for backwards compat; production settings.yaml enables
  SGOV with 1% reserve.
- broker.open_buy_notional(): None-vs-0.0 distinction so a failed order
  query reads as 'unknowable', never as 'no pending buys'.

Also pins QUANT_AGENT_MAX_RETRIES in the retry-deadline test (was sensitive
to ambient env overrides).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQkESoSTYx2bCy7WYnTPXR
2026-07-16 sell autopsy: 28/53 realized exits since 5/1 were EARLY (stock
>=5% higher within 20 days), $8.5k-16k left on the table; 5 trail-stop
fills missed avg +30.7% post-exit; LLY was trail-whipsawed twice
identically 4 weeks apart. The mechanical share of the leak gets code
guards (prompts alone demonstrably didn't hold):

- TRAIL_STOP noise-band clamp: a new stop inside 1.25×ATR14 of current
  price sits inside one day's normal range — rejected, old stop kept.
  A cited hard trigger bypasses (same philosophy as the same-day-trim
  gate). Unknowable ATR degrades open (no clamp).
- TRAIL_STOP ratchet cooldown: at most one accepted tighten per ~2
  trading days per symbol (the reviewer's >=1.02×old_stop min-bump rule
  made every accepted trail tighten >=2%; GE was ratcheted 325->350 in 8
  sessions on one flag). Hard triggers bypass.
- Live stop truth in position facts: prefer broker.get_current_stop_price
  over the stale-wide BUY-row stop (after any trail the reviewer saw a fat
  distance_to_stop and kept ratcheting), plus atr_pct /
  stop_distance_atrs so the reviewer reasons in vol units.
- Entry ATR floor (ExecutionStage): a BUY stop closer than 1×ATR14
  (computed from bars already fetched by research) is widened to 1×ATR;
  qty_by_risk sizes against the wider distance so per-trade $ risk is
  unchanged. No bars -> no floor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQkESoSTYx2bCy7WYnTPXR
… exit audit

RC4 (2026-07-16 forensics): the learning loop couldn't emit 'we were wrong
to sell' or 'we were wrong to skip':

- value_entry_missed was code-filtered OUT of the recurring-miss digest —
  the exact category evening uses for actionable entries (SNDK flagged 16×,
  ORCL 7×; PM was shown '(no recurring missed themes)' every run since 7/01
  while evening wrote 'zero execution — this is process failure').
- Misses were grouped by theme_if_any, LLM free text that never repeats
  verbatim (45 distinct themes, 0 recurring) — now grouped by SYMBOL, theme
  kept as annotation.
- Sell grades are scored by the LLM at t+1..t+3 with thesis-rationalization
  framing (32/33 'correct' while the tape had 53% of exits ≥5% higher within
  20d; grader error ~50% on checkable sells). A deterministic post-exit
  reality block (_build_post_exit_reality: trades × live prices, no LLM)
  now rides with the grade summary into the reviewer prompt, with a hard
  escalation line when over half of recent exits kept running. SWEEP_SELL
  and <2-day-old exits excluded.
- Reviewer prompt: to_stop is advisory distance, never a trigger (GS 5/18
  pre-empted its own stop); single-source bearish news on a >10% winner caps
  first-day action at REDUCE≤50% (AAPL 6/25); think in ATR units; documents
  the trail cooldown + noise-band enforcement from the exits commit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQkESoSTYx2bCy7WYnTPXR
…nswered

RC3 (2026-07-16 forensics): macro demanded 72-75% invested for three
months (risk-on the whole window); realized invested% averaged 39.2% and
declined monotonically 72.7%→15.4% over 13 weeks. Every layer shaved
sizes independently (PM on calibration, RM on R/R, evening tilt) and
nothing reconciled the compounded result against the target — ~$8.9k of
cash drag, 81% of the SPY shortfall.

- PMFacts gains macro_target_invested_pct + deployment_gap_pp; a >15pp
  under-target book renders a ⚠️ DEPLOYMENT GAP section the PM prompt now
  requires be answered in the cash_target step: close the gap this session
  OR name a checkable blocker per unfilled slot. 'Staying cautious' without
  a named blocker is explicitly ruled out. Does not override RM, caps, or
  drawdown-halving — it forces the gap to be addressed, not auto-filled.
- macro_exposure_deviation advisory is now direction-aware: for an UNDER-
  deployed book it tells RM NOT to scale the remaining BUYs down for this
  reason (the old symmetric 'consider scale_all_buys' amplified the drag
  it should have been correcting).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQkESoSTYx2bCy7WYnTPXR
… lane

RC2 (2026-07-16 forensics): when LLM latency inflates, morning dies at the
wrapper's outer timeout exactly at the PM→RM boundary — research (3 tech
chunks + news + macro) plus PM eat the budget and the kill lands as RM
starts. 61/61 PM BUY-proposal days between 6/30 and 7/15 were destroyed
this way (zero RM vetoes, 100% mechanical attrition), and every 30-min
retry tick re-burned the FULL research pipeline to die identically
(run-57b752bd: 'Constructor: 5 targets → 4 BUY' logged at 1196s of a
1200s budget). The relay outage that triggered it is fixed, but the
structural fragility — the only BUY-capable session being the first
casualty of any latency inflation — is not.

- src/decision_checkpoint.py: the plan is persisted (atomic tmp+rename)
  the moment DecisionStage produces decisions; ET-date-keyed, versioned,
  90-min max age.
- run_morning resume lane: after the FULL normal preamble (WAL drains,
  orphan sweep, coverage audit, stale-order cancel, force_delever, circuit
  breaker, fresh account snapshot), an unconsumed same-day checkpoint
  skips research+PM and re-enters at RiskStage — ~2 LLM calls instead of
  ~8. RM ALWAYS re-runs (no resume-past-RM variant, by design); the
  hard-risk filter re-runs on fresh cash/positions; ExecutionStage's 5%
  entry-staleness skip and pre-BUY daily-loss recheck guard stale prices.
- At-most-once: the checkpoint is marked consumed BEFORE ExecutionStage
  submits and on ANY RiskStage early-exit — an RM-rejected plan is never
  re-offered (that would be a veto bypass), and a kill mid-execution is
  owned by the existing BUY write-ahead orphan sweep, never by re-running
  the plan.
- Every checkpoint operation is best-effort: failures degrade to a normal
  full run, never a crashed session.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQkESoSTYx2bCy7WYnTPXR
…check

RC5 (2026-07-16 forensics): a 1-day incident became 13+ days because both
observability layers were blind to kills:
(a) timeout's SIGTERM/SIGKILL skips python's finally-block notifier — 13
    straight mornings died with ZERO Telegram pushes;
(b) evening's _expected_sessions_missing_today only checked that ANY run-
    row existed — partial runs (research logged, then killed) satisfied it,
    so the 🔴 banner never fired either.

- Wrapper: on violent deaths only (124/137/143 — the codes where python's
  own notifier cannot run) push a Telegram message directly from bash;
  ordinary non-zero exits stay python-owned to avoid duplicate pushes.
  TELEGRAM_DISABLED kill switch honored.
- Wrapper: optional HEALTHCHECKS_URL ping on success, /fail on failure —
  the external dead-man's switch CLAUDE.md has wishlisted since May; covers
  total host death and evening-not-firing, which no in-process check can.
- Dead-man check: two sharper probes when 'run-' rows exist — (a) research
  ran but portfolio_manager never logged (killed mid-research), (b) the
  decision checkpoint exists unconsumed (killed at the PM→RM boundary, the
  observed death mode). New db.agent_names_logged_on() supports the probe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQkESoSTYx2bCy7WYnTPXR
…park bookend

Two gaps caught in self-review before the adversarial pass:
- run_position_review computed the SGOV-free view but still passed raw
  positions into position facts / the reviewer / the action dispatcher —
  the reviewer would have hold-graded parked cash.
- park_excess was hooked into morning only; midday/close SELL proceeds
  would have sat unswept overnight. Both sessions now park at the bookend
  (emergency paths return earlier and deliberately skip parking).

Adds an end-to-end run_midday test pinning both behaviors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQkESoSTYx2bCy7WYnTPXR
…pass

A 34-agent review workflow (5 lenses × independent skeptic per finding)
confirmed 22 defects (7 claims refuted). Grouped fixes:

- RM sweep view (major): RiskStage passed raw positions to the RiskManager
  — the VETO layer saw parked SGOV as an 84%-of-book position while PM saw
  the same dollars as cash, desynchronizing the two risk layers with veto
  power on the corrupted view. RM/correlation/has_book now use the scrubbed
  list; the hard filter keeps RAW positions (it derives the parked-cash
  credit from finding the vehicle itself). Regression test added.
- Emergency×park oscillation (major): the post-review breach branch in
  run_position_review fell through to the park bookend — the system would
  buy SGOV with ~all equity minutes after force-selling everything, then
  the next intra tick would emergency-sell the fresh SGOV lot. Now returns
  emergency_sold immediately, mirroring the pre-review breaker.
- Ex-div false cooldown (major): ex-div adjustments write TRAIL_STOP rows
  (stop LOWERING) — they no longer count as a 'tighten' for the ratchet
  cooldown, so dividend names don't get spurious 2-day trail freezes.
- Resume-lane bars (major): the checkpoint deliberately omits symbols_bars;
  resume now rehydrates OHLCV for the plan's BUY symbols deterministically,
  restoring the entry ATR floor and the correlation advisory on resume.
- Checkpoint lifecycle: mark_consumed is fail-CLOSED (falls back to
  deleting the file — unlink survives ENOSPC — and returns bool); both
  emergency-liquidation exits consume the checkpoint and record a status
  marker; legit no_data mornings record status too.
- Dead-man accuracy: probes skip legitimately-PM-less mornings via the
  status marker; the notifier matches decorated 'morning (...)' entries so
  the sharpened diagnoses actually reach the 🔴 banner.
- Sweep hygiene: evening today_trades excludes SWEEP_* churn (limit 30→20
  after filter); post-exit reality skips the sweep symbol (emergency exits
  of SGOV are not decision-quality data); park_excess sizes qty against
  the LIMIT price so a padded fill can't overdraw a thin reserve.
- Entry ATR floor: after widening, recompute R/R vs the unchanged target
  and skip the BUY below 1.2 — RM approved the tight-stop geometry, not a
  collapsed one.
- PMFacts renders the OVER-deployment case instead of calling +40pp
  'within band'.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PQkESoSTYx2bCy7WYnTPXR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant